Hybrid programmable many-core device with on-chip interconnect

ABSTRACT

The present invention provides a hybrid programmable logic device which includes a programmable field programmable gate array logic fabric and a many-core distributed processing subsystem. The device integrates both a fabric of programmable logic elements and processors in the same device, i.e., the same chip. The programmable logic elements may be sized and arranged such that place and route tools can address the processors and logic elements as a homogenous routing fabric. The programmable logic elements may provide hardware acceleration functions to the processors that can be defined after the device is fabricated. The device may include scheduling circuitry that can schedule the transmission of data on horizontal and vertical connectors in the logic fabric to transmit data between the programmable logic elements and processor in an asynchronous manner.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of application Ser. No. 16/186,248,filed Nov. 9, 2018, entitled “Hybrid Programmable Many-Core Device withOn-Chip Interconnect,” which is a continuation of application Ser. No.15/823,353, filed Nov. 27, 2017, which issued as U.S. Pat. No.10,127,190 on Nov. 13, 2018, entitled “Hybrid Programmable Many-CoreDevice with On-Chip Interconnect,” which is a division of ApplicationNo. 15/288,569, filed Oct. 7, 2016, which issued as U.S. Pat. No.9,830,300 on Nov. 28, 2017, entitled “Hybrid Programmable Many-CoreDevice with On-Chip Interconnect,” which is a continuation ofapplication Ser. No. 13/804,419, filed Mar. 14, 2013, which issued asU.S. Pat. No. 9,471,537 on Oct. 18, 2016, entitled “Hybrid ProgrammableMany-Core Device with On-Chip Interconnect,” the disclosures of each ofwhich are hereby incorporated by reference herein in their entirety.

BACKGROUND OF THE INVENTION

The present invention relates to a hybrid programmable logic devicecontaining many programmable processors, dedicated function blocks, andprogrammable FPGA fabric. The present invention is particularly usefulfor network and packet processing, although it may be used in otherapplications.

Many-core (i.e., multiple core) devices have provided a way to increaseperformance of a device without incurring the cost of increasing clockspeeds. Many-core devices may include dedicated ASIC blocks for hardwarespecific functions, such as error control coding or cryptography. Theseblocks are often referred to as hardware accelerators.

SUMMARY OF THE INVENTION

The present invention provides a hybrid programmable logic device whichincludes a programmable field programmable gate array logic fabric(e.g., programmable logic elements) and a many-core distributedprocessing subsystem (e.g., many-core processors). It is noted that theterm “hybrid” refers to a device which integrates both a fabric ofprogrammable logic elements and processors in the same device, i.e., thesame chip. In the provided architecture, the programmable logic elementsmay provide hardware acceleration functions that are “late binding,”meaning that the specific functionality and logic architecture used toform a hardware acceleration function can be defined after the device isfabricated. For example, the programmable logic elements may bepartitioned or grouped into hardware accelerators via software thatprograms the hybrid programmable logic device at runtime. It is alsonoted that the term “device” refers to any embodiment or combination orembodiments of a hybrid programmable logic device described herein.

In certain embodiments, the processors may be integrated into theprogrammable logic fabric such that one or more physical dimensions ofthe processors, such as width, length, or height, are a multiple of thesame physical dimension (i.e., the corresponding length, width, orheight) or the programmable logic elements. This allows for efficientassembly of devices having different dimensions, with differing ratiosof processors to programmable logic elements or other logic blocks. Incertain embodiments, the processors may be “hardened,” meaning that theyconsist of fixed logic elements rather than programmable logic elements.This feature provides the integration of powerful many-core processorsinto a programmable logic fabric. In certain embodiments, the processorsmay be integrated into the fabric such that they are tiled in rows orcolumns. It is noted that the term “tiled” refers to arranging theprocessors such that they are interleaved or interspersed amongprogrammable logic elements. For example, the processors may be arrangedconsecutively in a two dimensional plane with the programmable logicelements such that there are no other programmable logic elementsbetween consecutive processors. This tiling of the processors may allowfor efficient fabrication and provisioning of interconnect networks forcommunicating between the processors, programmable logic elements, andI/O interfaces of the device.

In certain embodiments, the hybrid programmable logic device may includea data bus to move data between the processors, I/O interfaces, andmemory on and off chip. In certain embodiments, this data bus may be“hardened,” meaning that it consists of dedicated circuit componentsrather than circuit components that can be reserved for other uses onthe device. This hardened data bus allows for high performance datatransfer both internal and external to the device. In certainembodiments, this data bus may include horizontal and verticalconnectors. Certain groups of the horizontal and vertical connectors maybe wired to provide data to and from programmable logic elements, whileother groups may be wired to provide data to and from the processors. Incertain embodiments, the horizontal and vertical connectors in bothgroups may be in the same metal layers or layers. These same metallayers may be located above a layer in the interconnect stack in thedevice that contains the processors and programmable logic elements. Incertain embodiments, the horizontal and vertical connectors may be laidout such that their vertical pitch (e.g., vertical position in theinterconnect stack) is a multiple of the size of one or more groups ofprogrammable logic elements, processors, or both. Placing the horizontaland vertical connectors for both the processor and programmable logicelements in the same metal layer, and sizing them such that they agreewith the size of the programmable logic elements and processors, allowsfor place and route tools (e.g., software) to address the processors andlogic elements as a homogenous routing fabric. In other words, portionsof the connectors that route data to and from the processors may beaddressed in the same manner as the connectors that route data to andfrom the programmable logic elements.

In certain embodiments, the hybrid programmable logic device may includean ingress/egress processing block that receives data (e.g., in the formof data packets), and forwards that data to other elements of thedevice. In certain embodiments, the device may include a network on chipbus which is dedicated to routing and/or carrying the data to and fromthe ingress/egress processing block to other elements of the device. Theingress/egress processing block may be a hardened ASIC, or may be ablock built from the programmable logic elements on the device.

In certain embodiments, the hybrid programmable logic device may includescheduling circuitry that can schedule the transmission of data on thehorizontal and vertical connectors in the logic fabric that transmitdata between the programmable logic elements and processors. In certainembodiments, this scheduling circuitry may receive interrupt messagesaddressed to at least one of the processors, meaning that they containinformation that indicate that the payload of the data is meant to beprocessed by a particular processor or processors. In other embodiments,the scheduling circuitry may use other methods of asynchronouscommunication to send data between the programmable logic elements andthe processors. In certain embodiments, the scheduling circuitry mayidentify which of the processors are meant to process a particular data(e.g., a block or packet of data), and distribute the data to theidentified processors. By using interrupt messages or other asynchronousmethods of communication, the scheduling circuitry may allow theprocessors, the programmable logic elements, and the data bus to run atdifferent clock speeds, thus breaking any timing dependency between thenetwork bus and the processor clock speed.

In certain embodiments, the scheduling circuitry may include a programinstruction memory that can be reprogrammed during operation of theprocessors. Each processor may, via the scheduling circuitry, detect anew mode of operation based on information in a received data packet,halt operation in response to the detection of a new mode of operation(e.g., flush out all instructions in any pipeline in the processor) andreprogram the program instruction memory based on data received fromother elements in the device.

In certain embodiments, processors and programmable logic may be dividedinto partitions in order to manage power. For example, partitions of thehybrid programmable logic device may be computed. Each partition mayinclude a subset of the processors and programmable logic elements. Atleast one measure of processor demand may be computed, and based on thismeasure the power state of the processors and programmable logicelements in a partition may be altered. This feature may allow the powerconsumption of the device to be dynamically managed based on thethroughput to particular groups of processors and programmable logicelements.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other advantages of the invention will be apparent uponconsideration of the following detailed description, taken inconjunction with the accompanying drawings, in which like referencecharacters refer to like parts throughout, and in which:

FIG. 1A is a hybrid programmable logic device in accordance with anembodiment of the present invention;

FIG. 1B is an expanded view of the hybrid programmable logic devicecontaining processors adjacent to programmable logic elements inaccordance with an embodiment of the present invention;

FIG. 1C is an expanded view of the hybrid programmable logic devicecontaining an interface between the processors and programmable logicelements in accordance with an embodiment of the present invention;

FIG. 2A is an illustrative view of the data bus in the hybridprogrammable logic device in accordance with an embodiment of thepresent invention;

FIG. 2B is another illustrative view of the data bus in the hybridprogrammable logic device in accordance with an embodiment of thepresent invention;

FIG. 3 is a hybrid programmable logic device with a hardened data busand ingress/egress in accordance with an embodiment of the presentinvention;

FIG. 4A is a hybrid programmable logic device with a hardened data busand ingress/egress in accordance with an embodiment of the presentinvention;

FIG. 4B is a hybrid programmable logic device with a hardened data busand ingress/egress in accordance with an embodiment of the presentinvention;

FIG. 5 is a hybrid programmable logic device with a hardened data busand ingress/egress in accordance with yet another embodiment of thepresent invention;

FIG. 6 is scheduling circuitry in accordance with an embodiment of thepresent invention;

FIG. 7A is an illustration of a partitioned hybrid programmable logicdevice in accordance with an embodiment of the present invention;

FIG. 7B is an illustrative process for reducing power consumption of ahybrid programmable logic device in accordance with an embodiment of thepresent invention.

DETAILED DESCRIPTION

When designing a many-core device, the type, and number of acceleratorsthat are required for particular applications may be hard to predict. Todesign a device that addresses more than a single solution, it is commonto build in accelerators in the device that are unneeded, and thus wastearea on the device and consume excess power. It would therefore bedesirable to design a many-core device with programmable logictechnology such that the accelerators could be defined at a later stagein the development process (e.g., after deployment of the many-coredevice).

Further, when designing a many-core device, many agents may requestintermittent access to external memory. For example, when the many-coredevice receives a packet, the payload is split from the header and sentto external memory (or internal buffer) while the header is sent to aprocessing unit (e.g., a microprocessor or hardware block) for a largeamount of processing time (e.g., hundreds of clock cycles). Theresulting header, which is possibly modified, is then rejoined with itspayload and queued for forwarding off-device. When hundreds of agentsare sending and receiving data, coordinating access to memory may bedifficult. It would therefore be desirable to design a many-core devicewith a shared interconnect that allows for flexible routing (e.g.,dynamic routing as compared to static routing).

FIG. 1A is a hybrid programmable logic device 100 in accordance with anembodiment of the present invention. Device 100 includes regions ofprogrammable logic elements 130, and processors 150. General purposeinput/output circuitry 110 can include one or more generic pins that canbe controlled or programmed by external software at runtime. In certainembodiments, general purpose input/output circuitry 110 can beconfigured to input or output data, can read or write data, can be usedas IRQs for wakeup events, or can be used to transfer data from sourcesperipheral to device 100. High-speed serial interface 120 can facilitatethe transfer of information between external sources of data (notshown), programmable logic elements 130, and processors 150 usinghigh-speed transceiver blocks. For example, high-speed serial interface120 may interface with an Ethernet connection to receive packets ofinformation, process these packets with programmable logic elements 130and processors 150, and switch the packets to different physicalinterfaces. Exemplary circuitry for routing received information isdiscussed below in connection with FIG. 3. In certain embodiments,high-speed serial interface 120 may transfer data at much higher speedsthan general purpose input/output circuitry 110. For example, high-speedserial interface 120 may read and write data at a rate of tens ofGigabits per second, while general purpose input/output circuitry 110may write data at a rate hundreds of Megabits. In certain embodiments,high-speed serial interface 120 may operate at line-rate, meaning theaggregate throughput rate of device 100 (e.g., 100 Gigabits per secondacross multiple serial channels), while processors 150 and programmablelogic elements 130 operate at a distributed rate, meaning that theaggregate throughput of processing threads (and corresponding memoriesand hardware acceleration blocks used with those processing threads) isno worse than the minimum throughput for the target application ofdevice 100.

Programmable logic elements 130 can include any combination of logicgates and memory. In certain embodiments, these programmable logicelements may be grouped into logic array blocks (“LABs”), referring to aunit of programmable logic resources in devices provided by AlteraCorporation, of San Jose, Calif. However, the invention is applicable toprogrammable logic elements from any source. In certain embodiments, theprogrammable logic elements may be grouped into hardware accelerationblocks. Each hardware acceleration block may be designated to perform acertain type of hardware event on received data. In certain embodiments,the hardware acceleration blocks may be configurable such that the eventis tailored to that particular situation. For example, the hardwareacceleration blocks can accept parameters that further define thehardware event to be performed on a received data packet. Parametersused to configure a hardware acceleration block may, for example, begenerated by processors 150. Parameters can be transmitted to thehardware acceleration blocks through a data bus (not shown) thatincludes horizontal and vertical connectors that are connected to eachof the programmable logic elements 130 as well as processors 150. Incertain embodiments, programmable logic elements 130 may include anysuitable memory clusters, such as M20K memory clusters.

In certain embodiments, the programmable logic elements 130 may beconfigurable into different hardware acceleration blocks, after device100 has been fabricated and deployed (e.g., during runtime of device 100or through a remote update procedure). Thus, the hardware accelerationblocks made up of the programmable logic elements 130 may be latebinding, which allows device 100 to be versatile in any number ofapplications in any number of domains. For example, device 100 may beupdated to account for the latest error correction, video and imageprocessing, or data management standards. This is different fromcommercial Network Processing Units, pipelined processors, and ASICdevices that have both processor and fixed hardware acceleration blocks,as these devices do not allow for the hardware accelerators to bedefined after deployment.

As depicted schematically in FIG. 1A, there is more area on device 100devoted to programmable logic elements 130 than processors 150. This isbecause programmable logic elements 130 are cheaper than processors 150.In certain embodiments, programmable logic elements 130 may besubstantially similar to the programmable logic elements of the StratixV FPGA sold by Altera Corporation of San Jose, Calif. However, it shallbe understood that any programmable logic elements suitable for an FPGAor PLD may be used as programmable logic elements 130.

Processors 150 can include any suitable number of processors withmany-core designs. These many-core designs may be based onmicroprocessor IP by vendors such as ARM, MIPS, and Tensilica. Thismicroprocessor IP allows for the ability to create customized embeddedprocessors (e.g., removing floating-point units), and customizedinstruction set architectures. In certain embodiments, the processorsmay be “hardened,” meaning that they consist of fixed logic elementsrather than programmable logic elements. This feature provides theintegration of powerful many-core processors into a programmable logicfabric.

As depicted schematically in FIG. 1A, processors 150 are tiled in afour-column assembly structure in portions of the device 100 whereprogrammable logic elements 130 are not present. In other embodiments,fewer or greater columns of processors may be used. In certainembodiments, the number of columns or processors, and the number ofprocessors in each column may be more or less than depicted in FIG. 1A.The number of columns and number of processors 150 in each column maydepend on the processor architecture for each processor. For example,more processors may be tiled on device 100 when the instruction cacheand data cache of each processor is reduced. In one example, MIPS 34Kmulti-threaded processors with 16 KB of instruction cache and data cacheare used. In this example, when shrunk to a design target of 28 mm, eachprocessor consumes less than 0.25 square microns of silicon area on thedevice, which is equivalent to 30 Stratix V LABs. It shall be understoodthat processors 150 may alternatively laid out in rows on device 100.

In certain embodiments, processors 150 are multi-threaded.Multi-threaded processors provide advantages to applications where it iscommonly required to interface with an off-chip memory or lookup whichcould take multiple clock cycles of processing time, and thus increaselatency, or to hide the latency of accelerator processing. For example,by blocking one thread in a processor to this lookup task, the processoris able to proceed with performing other functions. In certainembodiments, the multi-threading may be achieved by time-slicingoperation on data received by the processors 150. In other embodiments,the multi-threading may be achieved by well-known operating systemmechanisms.

The tiled layout of processors 150 on the device allows for efficientfabrication and provisioning of interconnect networks for communicatingbetween the processors, programmable logic elements, and I/O interfacesof the device. This is because the programmable logic elements 130 andother elements of device 100 such as memory (not shown) are separatefrom processors 150. In addition, the tiled layout of processors 150allows for the construction of processors 150 and programmable logicelements 130 in the same metal layers in device 100, as well theconstruction of an interconnect stack containing a data bus.

FIG. 1B is an expanded view of the hybrid programmable logic device 100containing processors 150 adjacent to programmable logic elements 130 inaccordance with an embodiment of the present invention. Two processorsare depicted schematically in FIG. 1B as squares containing threecomponents—processor core 170, instruction cache 180, and data cache190. Also depicted schematically in FIG. 1B are programmable logicelements 130 that are grouped into LABs 132. In certain embodiments, thephysical size of a dimension of the processors 150, such as length,height, or width, is a multiple of the same dimension of theprogrammable logic elements 150, or LABs made up of programmable logicelements 150. For example, as depicted in FIG. 1B, the width of each ofthe two processors is equal to three LABs 132. This allows for efficientassembly of devices having different dimensions, with differing ratiosof processors to programmable logic elements or other logic blocks.

FIG. 1C is an expanded view of the hybrid programmable logic device 100containing an interface 161 between the processors 150 and programmablelogic elements 130 in accordance with an embodiment of the presentinvention. Interface 161 may be included between every LAB or row ofLABs and processors 150. Interface 161 may include LIM/LEIM circuitry162, interface logic 164, and interface port 166. LIM/LEIM circuitry 162refers to collections of programmable input muxes “LAB Input Mux” and“Logic Element Input Mux,” and may include any suitable number of LIMand LEIM multiplexors as commonly found in devices sold by AlteraCorporation of San Jose, Calif. Interface logic 164 may include anysuitable circuit components for buffering data to interface port 166.Interface port 166 may include any suitable circuitry for physicallydelivering signals to one of processors 150.

In certain embodiments, horizontal and vertical connectors 167 may alsobe included on device 100. As will be discussed below with respect toFIG. 3, horizontal and vertical connectors 167 may transfer data betweenprocessors 150, programmable logic elements 120, general input/outputinterfaces, and memory on and off chip. In certain embodiments,horizontal and vertical connectors may be wired to interface circuitry161 such that any signal from any of the programmable logic elements 130can be delivered to interface port 166 via LIM/LEIM circuitry 162 andinterface logic 164. For example, multiplexors within the LIM/LEIMcircuitry 162 may choose which signal drives any global wire in thehorizontal/vertical connectors 167 to connect any of the programmablelogic elements 130 with processors 150.

FIG. 2A is an illustrative view of the data bus 210 in the hybridprogrammable logic device 100 in accordance with an embodiment of thepresent invention. Although the wires of data bus 210 are shown abovethe LABs 215 in the vertical direction, in implementation they are partof an interconnect stack which is above the LABs (e.g., in a metal layerabove the metal layers containing the LABs and processors). Similar tohorizontal and vertical connectors 167, data bus 210 can transfer databetween processors 150, programmable logic elements 130, generalinput/output interfaces 110, and memory on and off the device 100 (FIG.1A). FIG. 2A illustratively highlights two wires of data bus 210 routedfrom LAB 216 of a particular spacing above LABs 215. In certainembodiments, the horizontal and vertical connectors of data bus 210 maybe laid out such that their vertical pitch (e.g., vertical position inthe interconnect stack, illustrated by ‘W’ in FIGS. 2A and 2B) is amultiple of the size (e.g., the vertical pitch) of one or more groups ofprogrammable logic elements, processors, or both.

FIG. 2B is another illustrative view of the data bus 210 in the hybridprogrammable logic device 100 in accordance with an embodiment of thepresent invention. As depicted schematically in FIG. 2B, processor 217is in place of LAB 216. Processor 217 may be part of a column of tiledprocessors in the device, such as one of tiled processors 150 (FIG. 1A).In certain embodiments, the wiring of data bus 210 has the same verticalpitch and follows the same pattern of the wiring of data bus 210 to LABs215 and LAB 216. In addition, in certain embodiments, both the wiringfrom processor 217 and LABs 215 and LAB 216 are in the same metal layerin the interconnect stack (e.g., in a metal layer above the metal layerscontaining the LABs and processors). By placing the horizontal andvertical connectors for both the processor and programmable logicelements in the same metal layer, and sizing them such that they agreewith the size (e.g., vertical pitch) of the programmable logic elementsand processors, allows for place and route tools (e.g., software) toaddress the processors and logic elements as a homogenous routingfabric. In other words, portions of the connectors that route data toand from the processors (such as processor 217) may be addressed in thesame manner as the connectors that route data to and from theprogrammable logic elements (such as the programmable logic elements inLABs 215 and LAB 216).

FIG. 3 is a hybrid programmable logic device 300 with a hardened databus 160 and ingress/egress processing block 136 in accordance with anembodiment of the present invention. FIG. 4A is the same hybridprogrammable logic device 300 with a different hardened data bus 160 inaccordance with an embodiment of the present invention. FIG. 4B is thesame hybrid programmable logic device 300 with another differenthardened data bus 160.

Device 300 also includes external memory 135 and embeddedternary-content addressable memory (CAM) memory 137. Hardened data bus160 may consist of dedicated circuit components that transfer data toand from processors 150, programmable logic elements 130, ingress/egressprocessing block 136, external memory 135, and embedded ternary-CAMmemory 137, rather than being reserved for other uses on the device. Incertain embodiments, hardened data bus 160 may be referred to as anetwork on chip interconnect. In such embodiments, hardened data bus 160may be dedicated to routing and/or carrying the data to and from theingress/egress processing block to other elements of the device.Hardened data bus 160 can allow for high performance data transfer bothinternal and external to the device. In certain embodiments, hardeneddata bus 160 may include a portion of horizontal and vertical connectors167 (FIG. 1C) and data bus 210 (FIGS. 2A and 2B). In other embodiments,hardened data bus 160 may be entirely separate from horizontal andvertical connectors 167.

Hardened data bus 160 may have a variety of topologies. For example,hardened data bus 160 may have a ring topology, a shared bus protocolsuch as AXI designed by ARM holdings of Cambridge, United Kingdom,interconnect technology from Sonics, Arteris or other third partycompanies, or Avalon interconnect designed by Altera Corporation of SanJose, Calif. In certain embodiments, hardened data bus 160 may behierarchical. For example, as depicted schematically in FIG. 3, hardeneddata bus 160 may be a 4-way division. In another example, as depictedschematically in FIG. 4A, hardened data bus 160 may be monolithic. Inyet another example, as depicted schematically in FIG. 4B, hardened databus may be segmented.

In certain embodiments, portions of hardened data bus 160 may beconnected or disconnected hierarchically by software that can programdevice 300. This software may include Quartus software design by AlteraCorporation of San Jose, Calif., or any other suitable software. Incertain embodiments, the bandwidth of hardened data bus 160 is designedto achieve the line-rate of a particular application. For example, ifhardened data bus 160 is 64 bits and operating at 1 GHz with 80%efficiency, it can provide 51 Gbps of bandwidth and be used to target a50 Gbps streaming video application. In another example, if hardeneddata bus 160 is 64 bits and operating at 1.5 GHz with 80% efficiency, itcan provide 150 Gbps of bandwidth and target a 100 Gb traffic switchapplication. In certain embodiments, separate hardened data buses may beadded to carry data for processor input and output, as LAB input andoutput.

In certain embodiments, hardened data bus 160 may be placed over (e.g.,in a metal layer above) or adjacent to processors 150 and programmablelogic elements 130. In this manner, hardened data bus 160 is minimallyinvasive to the assembly of device 300 (FIG. 3).

External memory 135 may include any suitable interface to externalmemory, such as DDR memory. External memory 135 can buffer data inapplications where device 300 is processing video data or packet data.In certain embodiments, embedded ternary-CAM memory 137 may include anysuitable block of content addressable memory, which is useful fornetworking applications. In certain embodiments, embedded ternary-CAMmemory 137 may interface with programmable logic elements 130 andprocessors 150 using circuitry substantially similar to interfacecircuitry 161 (FIG. 1C). In certain embodiments, ternary-CAM memory 137may be replaced by an SRAM or embedded SRAM packet buffer memory.

Ingress/egress processing block 136 can receive data and forward thatdata to other elements of device 300. For example, ingress/egressprocessing block 136 can receive data packets and provide protocoltermination or packet framing services to device 300. In certainembodiments, ingress/egress processing block 136 may forward receiveddata to hardened data bus 160. In certain embodiments, theingress/egress processing block may consist of hardened ASIC blocks.These blocks may be structured according to the ASIC blocks disclosed inU.S. Pat. No. 8,314,636, which is incorporated by reference herein inits entirety. In certain embodiments, ingress/egress processing blockmay be built from programmable logic elements 130.

FIG. 5 is a hybrid programmable logic device 500 with a hardened databus 160 and ingress/egress 137 in accordance with yet another embodimentof the present invention. Device 500 may be substantially similar todevice 100 (FIG. 1A) and device 300 (FIG. 3), except that a universalinterface bus 138 is integrated into device 500 for interfacing to 2.5Dor 3D memory, or other functions. As schematically illustrated in FIG.5, interface 138 can be connected to processors 150 and programmablelogic elements 130 via hardened data bus 160. In other embodiments,interface 138 can be connected to processors 150 and programmable logicelements 130 through general-purpose FPGA routing, such as horizontaland vertical connectors 167 (FIG. 1C). In certain embodiments, interface138 may be manufactured according to the design and methods described inU.S. patent application Ser. Nos. 13/350,662 and 13/620,126, which areincorporated by reference herein in their entirety.

FIG. 6 is scheduling circuitry 600 in accordance with an embodiment ofthe present invention. Scheduling circuitry 600 includes data bus 605,which can be substantially similar to horizontal and vertical connectors167 (FIG. 1C), or hardened data bus 160 (FIG. 3). Scheduling circuitry600 can schedule the transmission of data on the horizontal and verticalconnectors, hardened data bus, between programmable logic elements suchas programmable logic elements 130 and processor 630. In certainembodiments, processor 630 may be part of a column of tiled processors,such as those described with respect to processors 150 (FIG. 1A). Insuch embodiments, copies of scheduling circuitry 600 may be integratedwith every processor in processors 150.

Scheduling circuitry 600 includes bus interface 610 and threadingcircuitry 620. In certain embodiments, scheduling circuitry may alsoinclude processor 630 and cache 640. Processor 630 may be substantiallysimilar to processors 150 (FIG. 1A), and cache 640 may be substantiallysimilar to instruction cache 180 and data cache 190. In addition,processor 630 may be part of a tiled column or row of processors, suchas that discussed with respect to processors 150 (FIG. 1A). In suchembodiments, scheduling circuitry 600 may be implemented on the devicefor every processor in the tiled rows or columns of processors.

Horizontal and vertical connectors 605 may transmit data to and from businterface 610. Horizontal and vertical connectors 605 may be part of atleast a portion of horizontal and vertical connectors 167, hardened databus 160, or both. In certain embodiments, bus interface 210 may transmitdata to and from programmable logic elements on a device and processorson a device using horizontal and vertical connectors 605. For example,bus interface 210 may transmit data between programmable logic elements130 of device 100 and processor 630 using horizontal and verticalconnectors 167. In certain embodiments, the transmission of data betweenthe programmable logic elements and the processors may occur at a rateof speed different, or asynchronous from, the rate of speed of the clockof processor 630. For example, the speed of the operation of theprocessor may be faster or slower than the speed of operation of theprogrammable logic elements on the device, and the rate of speed atwhich data is transmitted between the programmable logic elements andprocessor using horizontal and vertical connectors 605 may be differentthan the speed of the clock of processor 630.

In such embodiments, bus interface 610 may provide mechanisms that allowdata to be transmitted on the horizontal and vertical connectors 605 ata rate asynchronous with respect to the speed of the clock of processor630. In certain embodiments, bus interface 610 may receive interruptmessages over horizontal and vertical connectors 605. In suchembodiments, these interrupt messages may be addressed to one or more ofthe processors on the device, meaning that they contain information thatindicate that the payload of the data associated with the messages ismeant to be processed by a particular processor or processors. Businterface circuitry 610 may transmit only the data associated withinterrupt messages that are addressed to processor 630, and ignore othermessages that are not addressed to processor 630. For example, businterface circuitry 610 may contain a predetermined or hardcoded value,either in software or in hardware associated with bus circuitry 610,that is the address of processor 630. Bus interface circuitry 610 mayperiodically or constantly monitor the messages transmitted overhorizontal and vertical connectors 605 for the address. When the addressis detected, bus interface circuitry 610 may buffer data associated withthe message addressed to processor 630, and transmit the data from thebuffer to processor 630 using threading circuitry 620. Threadingcircuitry 620 maintains one or more program counters associated with theexecution of various tasks in processor 630. In certain embodiments,threading circuitry 620 receives data from bus interface circuitry 610,and determines the appropriate time to send that data to processor 630based on one or more of the program counters that it maintains. In thismanner, scheduling circuitry 600 may allow the processors, programmablelogic elements, and the data bus of a device to operate at differentclock speeds, thus breaking any timing or clock dependency between thecomponents tied to the network bus and the processor clock speed.

In certain embodiments, any suitable methods of asynchronouscommunication may be used to send data between the programmable logicelements of the device and processor 630. In certain embodiments, thefunctionality of scheduling circuitry 600 may be achieved by groups oftwo or more processors rather than scheduling circuitry 600. Thesegroups of processors may identify which of the processors to send datareceived from the horizontal and vertical conductors of the device(i.e., transmitted from the programmable logic elements), and distributethe data to the identified processors. This identification may bedetermined similar to the monitoring procedure described above withrespect to bus interface circuitry 610.

In certain embodiments, scheduling circuitry 600 may include a programinstruction memory (not shown), which can include any suitablecombination of external memory. In certain embodiments, the instructionset of the processors on the device, such as processor 630, may bemodified during operation such that they are reprogrammed with a newinstruction set. This new instruction set may allow the device toachieve new or different functionality, such as program instructions fora hardware accelerator that may be used by the processors.

In such embodiments, the program instruction memory may detect a newmode of operation for one or more of the processors. For example, theprogram instruction memory may contain instructions for monitoring datapackets on the horizontal and vertical connectors 605. Certain datapackets, which can be identified by their packet header, may contain acontrol packet that contains instructions for reprogramming processor630 with new or updated functionality. Bus interface circuitry 610 maydetect these control packets by monitoring horizontal and verticalconnectors periodically or continuously to match the control packetheaders. In response to the detection of a new mode of operation, theoperation of processor 630 may be halted. For example, once businterface circuitry 610 detects a control packet header, it may bufferthe payload data of that packet and transfer the payload data tothreading circuitry 620. Threading circuitry 620 may then perform one ormore of the creation, execution, or insertion instructions in thethreads or pipeline of processor 630 that cause processor 630 to haltoperation, and flush out all pending instructions. Threading circuitry620 may then transfer the payload of the control packet that containsthe instructions for reprogramming processor 630 from a buffer in businterface circuitry 610 to processor 630. In certain embodiments,information in the header of the control packet may target a group ofprocessors, such as a group of processors in one or more tiled columnson the device. In such embodiments, the process described above mayoccur substantially in parallel for each of the targeted processors onthe device using each processor's associated bus interface circuitry.

In certain embodiments, the processors on the device may be divided intodifferent classes via partitioning. This partitioning may be physical(e.g., hard-coded in the processor), or virtual (e.g., assigned via arecord kept by scheduling software) FIG. 7A is an illustration of apartitioned hybrid programmable logic device 100 in accordance with anembodiment of the present invention. Device 100 may be substantiallysimilar to that described with respect to FIG. 1A. Device 100 may bepartitioned into a first partition 702, second partition 704, and thirdpartition 706. These partitions may be designated by place and routetools, or any suitable software. In certain embodiments, the partitionsmay be designated by a table that maps each of the processor's addressesor other unique identifiers with a partition number. In certainembodiments, the partitions of processors may also be associated with apartition of the programmable logic elements of device 100. It shall beunderstood that while FIG. 7A shows three partitions, the processors ondevice 100 may be divided into any suitable number of partitions in anysuitable geometry on the device.

Each partition 710, 720, and 730 may be associated with a differentclass of processor and programmable logic elements. In certainembodiments, these classes may be used to implement alternativeprocessing models via different types of processing. In suchembodiments, each partition may be associated with a differentprocessing or hardware acceleration function. For example, theprocessors associated with first partition 702 may be designated tohandle flow identification processing, the processors associated withsecond partition 704 may be designated to handle initial packetprocessing and distribution, and the processors associated with thirdpartition 706 may be designated to handle system or device-wide stateprocessing. In this manner, device 100 may be configured to handle anynumber of applications. The use of device 100 in packet processingapplications is further illustrated and discussed in U.S. patentapplication Ser. No. 13/804,419, titled “Mapping Network Applications ToA Hybrid Programmable Many-Core Device”, filed concurrently herewith,which is incorporated by reference herein in its entirety.

In certain embodiments, the different classes of processors andprogrammable logic elements may be used to manage power usage of device100. For example, processors and programmable logic elements associatedwith partitions of device 100 may be powered up and down dynamicallybased on the demand for the processing capabilities of the processorsand logic elements associated with each partition. This process isdescribed with respect to FIG. 7B below.

In certain embodiments, processors and programmable logic elementsassociated with one or more partitions of device 100 may be designatedas housekeeper processors. These housekeeper processors may implementfunctions such as power and clock management on behalf of device 100. Incertain embodiments, these housekeeper processors may be designated bythe user of place and route tools. In other embodiments, thesehousekeeper processors may be designated by device 100 automaticallyduring runtime of device 100.

FIG. 7B is an illustrative process 700 for reducing power consumption ofa hybrid programmable logic device, such as device 100 (FIGS. 1A and7A), in accordance with an embodiment of the present invention. Process700 begins at step 710. At step 710, at least two partitions of thedevice are computed, each partition including processors associated withprogrammable logic elements. Each of these two partitions may besubstantially similar to partitions 702, 704, and 706 discussed withrespect to FIG. 7A. In certain embodiments, the partitions may becomputed by place and route software before operation of the device. Inother embodiments, the partitions may be computed by scheduling softwareduring the operation of the device. This scheduling software may beexecuted on a host processor external to the device, or on designatedhousekeeper processors. In such embodiments, the scheduling software maychange the partitions dynamically during the operation of the device.For example, at a first time, the scheduling software may designate afirst partition associated with 40% of the processors and associatedprogrammable logic elements that are actively being used for hardwareacceleration functions, and a second partition associated with 60% ofthe processors and associated programmable logic elements that are notactively being used. Then, at a subsequent time, after more of theprocessors and programmable logic elements have become active, thescheduling software may designate a first partition associated with 55%of the processors and programmable logic elements that are activelybeing used for hardware acceleration, and a second partition associatedwith 45% of the processors and associated programmable logic elementsthat are not actively being used. In this manner, the device canactively adjust its power usage based on throughput of the device. Incertain embodiments, the partitions may be associated with a “top half”and “bottom half” of the processors and associated programmable logicelements. In such embodiments, scheduling software may designate thepartition associated with the “bottom half” of the resources in thedevice in a power savings mode while the partition associated with the“top half” of the resources in the device may be designated for anactive mode. In certain embodiments, the partitions may be computed bydetermining what hardware acceleration functions are being executed by aprocessor or group of processors, and then designating those processorsand the programmable logic elements associated with executing aparticular hardware acceleration function as a single partition.

Process 700 may proceed to step 720. At step 720, at least one measureof processor demand is computed. In certain embodiments, these measuresof processor demand may be computed for each partition computed at step710. In certain embodiments, a measure of processor demand may becomputed by the scheduling software. In certain embodiments, the measureof processor demand may be computed by determining the number ofreceived data packets that are waiting (e.g., in a queue) to beprocessed by a processor or a group of processors (e.g., all of theprocessors in a particular partition). In one embodiment, if a processoror group of processors have not received data packets for apredetermined period of time (e.g., because the processors andassociated programmable logic elements perform a hardware accelerationfunction that is not being used), the processor or group of processorsmay be designated as unneeded, and marked for powering down as will bediscussed with respect to step 730 below. In this manner, processorswhich are no longer needed for their processing power are figuratively“laid off” from performing their job on device 100. In the sameembodiment, if data packets are subsequently received for the processoror group of processors marked as idle, those data packets may be queuedand the processor or processors may be designated as active and markedfor powering on as will be discussed with respect to step 730 below. Inthis manner, idle processors which are needed again are figurativelyhired back to performing their job on device 100. In another embodiment,if a processor or group of processors have received a number of datapackets below a predetermined threshold, the processor or group ofprocessors may be designated as unneeded, and marked for powering downas will be discussed with respect to step 730 below.

Process 700 may then proceed to step 730. At step 730, the power stateof the processors and associated programmable logic blocks in thepartitions may be altered based on the measure of processor demandcomputed at step 720. In certain embodiments, the power state of apartition may be altered by powering down the resources of the deviceallocated to that partition. For example, if processor demand iscalculated for a particular partition as being less than a thresholdnumber of received data packets, then the processors or groups ofprocessors and associated programmable logic elements of that partitionmay be powered down. In certain embodiments, resources within apartition may be powered down by disabling the clock signal provided tothose resources. In certain embodiments, the power state of a partitionmay be altered by powering up the resources of a partition that waspreviously powered down or is dormant. For example, if the devicedetermines that the resources in a particular partition that is powereddown are needed to execute a new or scheduled hardware accelerationfunction, or that data packets are currently being received that callfor resources associated with a hardware acceleration function that werepreviously powered down, then the resources within that partition may bepowered up. In certain embodiments, resources within a partition may bepowered up by enabling the clock signal provided to those resources.

It will be understood that the foregoing is only illustrative of theprinciples of the invention, and that various modifications can be madeby those skilled in the art without departing from the scope and spiritof the invention. One skilled in the art will appreciate that thepresent invention can be practiced by other than the describedembodiments, which are presented for purposes of illustration and not oflimitation, and the present invention is limited only by the claims thatfollow.

What is claimed is:
 1. A hybrid data processing device, comprising: aplurality of regions respectively comprising: a hardened processorconfigurable to operate at a first clock rate, wherein the hardenedprocessor is communicatively coupled to a respective programmable logicelement operating at a second clock rate different from the first clockrate; and a plurality of connectors configurable to transfer databetween the hardened processor and the respective programmable logicelement; a hardened data bus that enables static routing between thehardened processor and the respective programmable logic element; and aninterface configurable to perform at least one synchronization functionto convert the data from the first clock rate to the second clock rate.2. The hybrid data processing device of claim 1, wherein the hardenedprocessor comprises an instruction memory and a data memory.
 3. Thehybrid data processing device of claim 1, wherein the hardened processoris configurable to perform floating-point precision operations.
 4. Thehybrid data processing device of claim 1, wherein the hardened processoris configurable as a hardware accelerator to accelerate a hardwarefunction.
 5. The hybrid data processing device of claim 1, wherein theplurality of connectors comprises horizontal and vertical connectors. 6.The hybrid data processing device of claim 1, wherein the plurality ofconnectors is configurable to form a network-on-chip.
 7. The hybrid dataprocessing device of claim 1, wherein the hardened processor comprises aprocessor core.
 8. The hybrid data processing device of claim 1, whereinthe plurality of connectors is compatible with an Advanced eXtensibleInterface (AXI) bus protocol.
 9. The hybrid data processing device ofclaim 1, wherein the hardened processor is configurable for parallelprocessing.
 10. A data processing system comprising a programmablecircuit, wherein the data processing system comprises: a plurality ofregions comprising: a first region disposed vertically adjacent to asecond region; the second region disposed vertically adjacent to thefirst region and a third region; the third region disposed verticallyadjacent to the second region, wherein at least one region of theplurality of regions comprises: a hard processor communicatively coupledto programmable logic, wherein the hard processor is configurable tooperate at a first clock rate and receive data from the programmablelogic, wherein the programmable logic is configurable to operate at asecond clock rate different from the first clock rate; and a pluralityof interconnections configurable to transfer data between the hardprocessor and the programmable logic; a hard data bus that enablesstatic routing between the hard processor and the programmable logic;and an interface configurable to perform at least one synchronizationfunction to convert the data from the first clock rate to the secondclock rate.
 11. The data processing system of claim 10, wherein the hardprocessor of the at least one region is configurable to executeinstructions stored by a program memory associated with the at least oneregion.
 12. The data processing system of claim 10, wherein theplurality of interconnections comprises horizontal and verticalinterconnections.
 13. The data processing system of claim 10, whereinthe hard processor is configurable as a hardware accelerator toaccelerate a hardware function.
 14. The data processing system of claim10, wherein the data processing system is configurable to performnetwork and packet processing.
 15. The data processing system of claim10, wherein the plurality of interconnections is configurable to form anetwork-on-chip.
 16. A programmable logic device comprising: a pluralityof regions comprising: a first region comprising: a first processorcommunicatively coupled to programmable logic; and a first plurality ofinterconnections configurable to transfer first data between respectiveregions of the plurality of regions; a second region comprising: asecond processor communicatively coupled to the programmable logic; anda second plurality of interconnections configurable to transfer seconddata between the respective regions of the plurality of regions; and ahard data bus that enables static routing between: the first processorand the programmable logic; the second processor and the programmablelogic; or both; and an interface configurable to perform at least onesynchronization function to convert the first data or the second datafrom a first clock rate to a second clock rate different from the firstclock rate, wherein the first clock rate is associated with the firstprocessor or the second processor, and wherein the second clock rate isassociated with the programmable logic.
 17. The programmable logicdevice of claim 16, wherein the first processor, the second processor,or both, comprises a processor core.
 18. The programmable logic deviceof claim 16, wherein the first processor, the second processor, or both,is configurable to execute instructions stored by a program memory. 19.The programmable logic device of claim 16, wherein the first processor,the second processor, or both, is configurable to perform floating-pointprecision operations.
 20. The programmable logic device of claim 16,wherein the programmable logic device is configurable to perform networkand packet processing.